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collection strategy and implementation plan is discussed. The 
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contains the current Office of Special Education and Rehabilitation 
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INTRODUCTION 



OVERVIEW 



This material is taken from the final report of a twelve-month 
study performed by Berkeley Planning Associates (BPA) for the U.S. 
Department of Education/Of f ice of Special Education and Rehabilitation 
Services (OSERS) to develop performance measures for supported employ- 
ment programs. 1 The term "supported employment" refers to a broad 
family of local programs and state-level projects designed to establish 
or expand supported employment opportunities for individuals with severe 
disabilities. The supported employment concept includes the provision 
of ongoing publicly — funded support services that participants need in 
order to sustain employment (hence the name "supported employment"). 

As defined by the five-year National Supported Employment 
Demonstration program implemented by OSERS in September 1985, supported 

employment consists of the creation or expansion of compensated, 

2 

meaningful work opportunities for individuals with severe disabilities. 
These work opportunities are to be created in job settings where dis- 
abled workers have opportunities for social interaction with nondisabled 
workers or the general public, and are intended to offer a significant 
number of hours of compensated work each week* The National Supported 
Employment Demonstration offers states additional financial resources to 
be used for "system transformation" — i.e. to transform existing day 
activity and work activity programs into systems that emphasize the 
opportunity for individuals to realize their potential for productive 
work in integrated settings. In many states, suppor' ed employment 
activities are also increasing in response to local initiative by pro- 
jects expanding options within local service systems (with or without 
formal state plans for system transformation, and with or without 
federal incentive funding). 

It has been the intent of this BPA study to help clarify supported 
employment program objectives and identify performance measures that can 
be used at a variety of different levels — federal, state, and local — 
to document the practices and achiev ement s of supported employment 
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programs as they continue to receive greater attention and are provided 
with a greater share of program resources throughout the nation* 

The activities carried out under the BPA study were framed as 
activities to "seek consensus" across a large number of actors involved 
in supported employment activities throughout the United States. One of 
the reasons for the emphasis on seeking consensus is that supported 
employment represents a loose envelope of closely-related and like- 
minded program efforts, rathei than a program with a single clearly 
defined identity supported by a single legislative mandete, a single 
funding source, or a single agency or organizational setting, Ir4 such 
an environment, it is likely that any efforts to generate summary data 
on program accomplishments will be the result of volixntary cooperation 
rather than hierarchically imposed data collection or reporting require- 
ments. The second reason for emphasizing consensus is the need to 
arrive at a core group of objectives and performance measures that 
adequately describe the common elements and intentions of supported 
employment programs that vary widely in their details of operation. At 
the heart of this study has been a recognition of, and a respect for, 
the diversity of supported employment approaches — ranging from indivi- 
dual placement models to mobile work crews to work enclaves within 
industry — as well as the diversity of program participants, funding 
sources, and agency roles. Thus, rather than being imposed on a homo- 
geneous set of programs or being viewed as a way to increase program 
uniformity, the data items and performance measures discussed in this 
report are attempts to identify and reflect movement toward common 
goals, objectives and desired outcomes by a very diverse universe of 
supported employment projects. 

The potential users of information about the ac '-nments of 

supported employment include a variety of actors, including: 

• federal and state policy m akers , interested in whether 
the supported employment initiative is accomplishing its 
goals; 

• state and local ad m inistrators ^ interested in whether 
system transformation is occurring, and if so, in what 
are its related cost and service impacts; 



• progra m operators, interested in tracking project 
resources and services rovided to participants, as well 
as the outcomes of services for participants, and in 
refining service designs to maximize project 
effectiveness : 

• consumer groups^ and individual parents and participants , 
interested in expanding com muni ty-ba se d employment 
options and in choosing the best program for themselves 
or their children with disabilities; 

• employers , who may be considering whether to hire a 
supported employment participant, or whether to sponsor a 
supported employment group work site ; and 

• social sc ient ists , interested in tracing the net impacts 
of the investment in supported employment from the 
participant and ti .payer perspectives, as well as in 
analyzing the factors influencing project effectiveness. 

Each of thase individuals or groups is interested in reviewing the 
accomplishments of supported employment efforts. Some users are 
primarily interested in outcomes at the local project level; others are 
interested in local system or state system outcomes, or in aggregate 
federal statistics. Each of these different information users has a 
stake, then, in the collection and reporting of data that will address 
their concerns. 

After expl oring the degree of agreement about supported empl oy ment 
objectives with representatives of a variety of program models and 
s level perspectives (e.g., individual projf local system, state 

nd federal perspectives) v 11 as ^arche s on and propo- 
viie supported employment coi. v^pt, we were able to identify a 
remarkable degree of consensus regarding six broad areas of desired 
program perf orm ance : 

• Meaningful Work; 

• Compensation; 

• Ongoing Support; 

• Worksite Integration and Community Participation; 
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0 Quality of Life; and 
• Community Change, 

Each of these goal areas, or domains, stands for a common desired 
program obj ec t ive or a set of related obj ec t ives of supported em pi oy~ 
ment. These performance domains became the initial framev-ork for the 
development of sets of candidate data items and potential performance 
measures to measure sup ported employment pr ogr am accomplishments. 
Ultimately, the performance domains have been expanded into eight data 
collection and measurement topics: 



• 


Measures 


of 


Employment Outcomes 


• 


Measures 


of 


Qual ity of Employment 


• 


Measures 


of 


Worksite Integration 


• 


Measures 


of 


Ongoing Support 




Measures 


of 


Quality of Life 


• 


Measures 


of 


Participant Characteristics 


• 


Measures 


of 


Systems Change 


• 


Measures 


of 


Program Cost s 



The basic building blocks required for analysis of most of these 
topics will be the collection and recording of basic client-level data 
by supported employment projects around the country on (1) the partici- 
pants, (2) the services provided, and (3) the outcomes experienced by 
supported employment clients. Additionally, proj ec t~level data on 
program costs and service models will be critical to most analysis 
efforts. Thus, it is our hope that the collection of core set of 
Cli nt— level data on an ongc^ng . * v ^ .ut be* viewed as a part 

trs^; .1^ . cost of "Cw-ing uusines s a supported employment service 
^ider. However, because c ollection does have clear costs associated 
with it, the data designated as "core data" clearly have to be keep to a 
minimum. 

The purpose of this study has been two-fold: (1) to facilitate 
discussion and encouraf>e the development of consensus about the most 
important core data items and performance measures which, it is hoped, 
will come to be considered the bare minimum for data collection and 
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reporting for the national supported employment initiative as a whol*^ 
and for its constituent state and local projects, and (2) to establish 
conceptual framework for the ^ollect and analysis of additional data 
to address sur* " _ i.y pc xormance questions of interest to a wide 

audience. Some of these additional or "supplementary" data elements may 
be feasible and appropriate for data collection by some or most projects 
on an ongoing basis for all supported employment participants. Others 
may be feasible or appropriate for data collection by some projects on a 
periodic basii? (e.g., one month out of every twelve) or for a random 
sample of all participants (i.e., a special study sample). The 
collection of still other data elements is clearly beyond the resources 
and/or capacity of local projects, and would require special research 
funding and/or special data collection efforts. 
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NOTE 



This material has been extracted from Berkeley Planning Asso- 
ciates' full Final Report to the U.S. Department of Education/Office of 
Special Education and Rehabilitation Services (OSERS) titled Development 
of Perf ormance Measures for Supported Employment Program s» where it 
constitutes Part II: Establishing Consensus About Recommended Data 
Items and Developing a Data Collection Strategy . The full Final Report 
includes an expanded discussion of the conceptual framework on which 
performance measures for supported employment must be based (in Part I: 
Establishing Consensus About Program Objectives and the Role of Perfor- 
mance MeasurementT » and a discussion of f uture research priorities to 
develop additional measures and to address supplementary evaluation 

questions (in Part III: Developing a Research Ag enda) . Readers 

interested in the full text of BPA's Final Report should contact the 
Publications Coordinator* Berkeley Planning Associates, 3200 Adeline 
Street. Berkeley. CA 9A703. Telephone (A15) 652-0999. 

2For a more detailed definition, developed by OSERS for designing 
the National Supported Employment Demonstration projects, sec 
Appendix A. 
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I. IDENTIFYING RECOMMENDED DATA ITEMS AND MEASURES 
INTRQDU CTION 

While the introduction to this material provided an overview of the 
conceptual framework within which the study evolved, the following 
section presents a practical framework for implementing common ongoing 
measures of supported employment efforts. These common measures fall 
within the eight data collection and measurement topics discussed 
previously : 

• employment outcomes: 

• quality of employment; 

• opportunities for worksite integration: 

• ongoing support ; 

• quality of life; 

• participant characteristics: 

• systems change; and 

• supported employment costs. 

Within each topic or domai:^, the BPA study team used the wide range of 
information gathered, through the literature and discussions with 
informed respondents, to identify which data items and measures might 
best be considered as: 

• universal measures that provide information on 
an ongoing basis, including data necessary for ongoing 
program monitoring and for summarizing performance at 
state and federal levels. These are also measures that 
would be most useful if implemented across projects 
and/or states; 

• supplementary , or additional measures that need not be 
the same across states and projects. These may include 
particular areas that a project would want to monitor for 
itself on an ongoing basis, as well as measures that 
reflect locil and/or project model pecularities; and 
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• evaluation^ or periodic measures or special studies that 
would be conducted for a representative sample^ 
occasionally, or only after the program has demonstrated 
longevity . 

Using these criteria, the study team presented a potential set of 
core, supplementary, and evaluation measures to 35 individuals from 
around the country for discussion and consensus building during a day 
and a half seminar. These individuals represented federal agencies, 
state and local public and private agencies from the fields of develop- 
mental disabilities, vocational rehabilitation, and special education, 
as well as educational institutions and research centers. The National 
Consensus Seminar occurred on May 20-21, 1986 in Washington, DC^ and was 
structured to (1) generate response to the potential measures; 
(2) create an opportunity f or a diverse group of involved persons to 
explore the extent of theoretical agreement and consistency in practice 
on program goals, implementation practices, and measures of program 
accomplishment; (3) allow federal, state, and local representatives to 
exchange information to develop an understanding of the dimensions of 
supported employment as it emerges around the country; and (A) establish 
communication links between program operators, researchers, and policy 
makerw During the seminar, consensus-building focused largely on 
individual proposed data items rather than on the aggregate performance 
measures that might be constructed from them. The consensus-building 
process centered on those measures considered by the group to be core or 
universal • 

This chapter reflects the group's work during this consensus- 
building process through tables displaying the set of core consensus 
measures and a set of supplementary or additional measures for each 
domain. In addition, each section attempts to brief ly address : 

• Why is this domain important to measure/what do we want 
to know? 

• What are the constraints in measuring this domain? and 

• What are the implications for collecting data in terms of 
the consistency and timing of data to be collected? 
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The lists of core consensus m- nnree represent a set of data items 
and ' •'nsures that would ideally >^r» imi trmented by all supported employ- 
mc. -r gr in a consistant way. While these consensus measures do 
not mean ^ ly a federal ii.Lcntion to mandate standardized data 

coll'^ tion, tiiey do represent a minisial set of agreed-upon measures and 
a ?t!£l ""'.itr froin thicsc sctivs in the fisld sbciit ^hat ths s^ippcrtsd 
employment program might hold itself accountable for and/or what 
describes the program's intent. In addition^ areas were identified 
where further research will be needed to develop measures and 
methodologiet . 

The supplemental V sets include measures that would entail choices 
made at project and/or b*:ate discretion about what particular informa- 
tion would be useful to collect. The specific measures included are 
suggestions or representations of ways to expand the core set of ongoing 
measures. Other measures which reflect project or community-specific 
concerns could certainly be added to the list of possibilities, along 
with special studies or evaluation topics. Chapter II discusses the 
various concerns which erise when attempting to develop a data 
collection strategy and implementation plan for ongoing performance 
monitoring. 

EMPLOYMENT ODTCXmES 

The key feature of supported employment that distinguishes it from 
day activity programs is paid employment. Employment exists when an 
individual's activities create goods and services that have economic 
value, and when he or she receives payment for work from an employer or 
customer. Therefore, the first step in measuring performance and 
perhaps the most important measure of success of supported employment 
programs is assessing the extent to which employment outcomes are 
achieved. These outcomes can be organized into four general dimensions: 
(1) actual placement/employment; (2) stability of that employment; 
(3) earnings; and (4) hours worked. These areas were identified as the 
key aspects of employment outcomes during the study, and are reflected 
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rifturtt lA 
BMFLOTMEHT OUTOOMES 
Core CoQSeDSUA DAta Xtcu and Reconaended MeaFiirec 



ConccnBug Dat a Items 



Recommended Core Measures 



• Enrolled in SE program 



• total # of participants served (active enrolleos) during 
reporting period 



• Obtained paid employment 



• # and X of participants in paid onployaent (at any time 
during reporting period) 



c Achieved SE outcome (current OSEKS 
definition) 



• # and Z of participants vho have achieved supported 
employment outcomer (current OSERS definition) 

• X of participants in paid employment who have achieved 
aupported employment (current OSERS definition) 



• Hourly wage at placement 



• mean hourly wage at placement 



• Hourly vage at end of reporting period 



• mean hourly supported employment vage (mean vage for all 
participants vho have achieved supported employment out~ 
comes) 

• mean hourly vage for all participants in paid productive 
work (including participants vho have not [yet] achieved 
aupported employment outcomes) 

• mean monthly eamingc for all participants in paid pro- 
ductive vork (including participants vho have not [yet] 
achieved aupported employment outcomes) 

• X of employed participants/participants achieving sup- 
ported employment earning minimum wage or above 



• Received fringe benefits: health 
coverage 

— yes 

— no 



• mean monthly supported employment earnings (mean monthly 
earnings for all participants vho have achieved supported 
employment outcomes) 

• total earnings across all participants vho have achieved 
supported employment outcomes for reporting period 

• total earnings across all participants (including those 
who have not yet achieved supported employment outcomes) 
for reporting period 



• Total hours worked during reporting 
period 



• mean hours worked per week in supported employment 



• # weeks employed since enrollment, for 
participants with SE outcomes 



• mean # weeks employed since enrollment for current pax 
ticipants with SE outcomes 



• # weeks employed during reporting 
period, for participants with SE 
outcomes 



• mean # weeks employed during reporting period for current 
participants with SE outcomes 



• # weeks employed with present employer, 
for participants with S£ outcomes 



• mean length of time (# weeks) with present employer for 
participants with SE outcome 
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in the core consensus data items and recommended measures listed in 
Figure lA. 

Measuring these dimensions will enable us to answer the questions: 
To what extent is the participant in supported employment involved in 
paid productive work, ho^ paid and h^w productive is the work, and is 
that work retained over a period of time? In additd on, this information 
will be an integral part of an internal assessment of project 
performance as well as providing bench marks against which to compare 
outcomes across projects. These comparisons will facilitate the 
identification of; (1) differences among supported employment projects; 

(2) differences between supported employment and other programs; 

(3) "best practices," or exemplary methods for achieving best outcomes; 
and (4) a long range picture of the supported employment program as it 
develops. 

As evidenced in Figure lA, the issue of defining who is a 
participant in supported employment is central to the consistent 
measurement of employment outcomes. The National Consensus effort 
resulted in a determination to collect data on all supported employment 
participants, defined as active enrollces , with certain meas ^s 
designed specifically to capture outcomes for those who fall within or 
achieve the current OSERS supported employment guidelines (i.e.. working 
at least 20 hours per week, in a group of eight or fewer disabled 
workers, and receiving publicly-supported, on-the-job support). Thus, 
the consensus items reflect a decision about questions of definition and 
priorities — decisions which will be necessary to make on a broader 
scale to promote the greatest possible consistency in data collection. 
Other National Consensus decisions which are evident in the core data 
items are: 

• an agreement to measure both hourly wage (to capt ure 
individual client level progress and changes in produc- 
tivity) and total monthly earnings over the reporting 
period — collected at the client level on a monthly 
basis and reported either monthly or quarterly on an 
aggregate level; and 

• an agreement to collect both total hours and number of 



ERIC 




12 



weeks worked during the reporting period for participants 
with SE Outcomes (as defined by OSERS). 

Measures of - 2Gks worked for participants who have not (yet) 
achieved supported employment (as defined by OSERS) were determined to 
be supplementary (see Figure IB). In addition, measures that looked at 
changes in earnings and employment status over time (by quarters) were 
also considered supplementary, as were measures of fringe benefits other 
than health coverage and more detailed information about hours worked 
per week. It is interesting to note the consensus reached about 
measuring hours and weeks worked in addition to earnings measures. 
Participation in work on a regular and sustained basis is valued as a 
goal in and of itself, independent of the wages and production yielded 
by that activity* 

The potential for collecting consistent employment outcome data 
across states and projects seems high, given an effort to utilize common 
definitions. Projects and states must also consider implementation 
issues. Much of this information can be collected at the client level 
relatively easily, and then aggregated at the project and state level. 
It nay be somewhat more difficult to collect client level data on fringe 
benefits, but measuring these at the employer level might not accurately 
represent whether or not participants actually receive fringe benef its. 
Ultimately, it is critical to look at all employment outcomes of 
supported employment in light of its goal for "normalized" employment. 
This will mean considering the data in the context of nondisabled 
coworkers and/ or the standards of that particular industry. 

QUALITY OF KMPLOTMENT 

One of the goals of the supported employment initiative is to offer 
participants an opportunity to shift from the often monotonous tasks 
performed in sheltered employment to productive and meaningful work 
involving variety in tasks, the opportunity to acquire new skills, 
increased job satisfaction and security, and employment mobility. Some 
programs have goals that even go further than placing clients in "real" 
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Fiaur* IB 

BMFLOYKEKX OOTOOKES 
Additional or SuppleseDtary Data Itcns and Other Possible Heasures 



Additional or Supplementary 
Data Items 


Other Forcible Measures 


• 


Received fringe benefits: sick leave, 
vacation ttmie* otber 

— yes 

— no 


• 


T of employed participants receiving: sick leave, vaca- 
tion leTve, other fringe benefits 


• 


Value of fringe benefits as a percen*- 
tage of earnings 


• 


mean fringe benefit rate (value of fringe benefits as per* 
centage of earnings) 


• 


Mean hours worked per veek during 
period employed 


• 


# of supported employment participwnts working: 

— 1-10 I T vs r .r week 

— 11-20 houj-b per week 

— 21-30 hours per week 

— 31-AO hours per week 


• 


Total hours spent in day program or 
work activity other than supported 
csployment during reporting period 




Z of total day program hours spent in supported employ- 
m'-'.it 


• 


# weeks employed since enrollment 
(for participants who have not [yet] 
achieved supported employment out- 
comes y 


• 


mean # weeks employed since enrollment (for ctirrent par- 
ticipants who have not [yet] achieved SE outcomes) 


• 


# weeks employed during reporting 
period (for participants who have 


• 


mean # weeks employed during reporting period (for par- 
ticipants who have not [yet] achieved SE outcomes 


• 


# weeks employed with present employer 
(for participants who have not [yet] 
achieved SE outcomes) 


• 


mean length of time (# weeks) with present employer (for 
participants who have not [yet] achieved SE outcomes) 


• 


Z weeks employed since first placement 
(for all participants) 


• 


mecin Z weeks employed since placement (for all partici- 
pants) 


• 


Duration of program participation 
— # quarters in program 


• 
a 


mean # quarters in program for current participants 
mean # quarters in program for terminees 


• 


Employment status at end of 2nd quarter, 
Ath quarter* 8th quarter. 12th quarter. 
etc»» after enrollment (based on con- 
sensus data items) 


• 


number and Z of total program participants employed at 
end of 2nd. Ath. 12th quarter after enrollment (a par- 
ticipant is included in this measure if the j'eporting 
period is his or her 2nd. Ath. 8th. 12th quarter after 
enrollment) 


• 


Earnings during 2nd. Ath. 8th. 12th 
qiiwter after enrollment (based on 
core data items) 


a 


mean monthly earnings during Ath. 8th, or 12th quarter 
after enrollment (a participant is included in this 
measure if the reporting period is his or her Ath. 8th, 
or 12th quarter after enrollment) 
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jobs in the community and have given priority to finding "non- 
traditional" placements for individuals with severe disabilities, as 
alternatives to what some consider to be stereotypical placements. 
ThuSg in order to measure the performance of supported employment 
efforts, the first step is to collect information about the nature of 
the work itself. 

In addition to the employment outcome measures described 
previously, most of the existing measurement systems reviewed and 
analyzed by the BPA study team included some information about the 
nat ure and adequacy of the w ork placement. While this was of ten 
informal and descriptive in nature, a number of potentially useful 
examples of more formal procedures for collecting and reporting data 
about work placement characteristics exist. Much more work is needed, 
however, to develop measures that can address the full range of impor- 
tant questions about the quality of the employment situation including: 

• What is the nature of the work — is it an improvement 
over "make-work" in sheltered settings? Is it real work? 
Is it meaningful work? Is there a good match between the 
worker and the job tasks? Is stereotyping occurring? 

• What is the quality of the work environment — are appro- 
priate job accommodations being made? What is the 
quality of the physical environment — is there enough 
light and air? What is the quality of the interactions 
between workers and supervisors — are there appropriate 
avenues f or communication and exchange of inf ormation? 

• Why do supported employment participants leave their jobs 
and where do they go after termination — are supported 
workers leaving job situations for reasons similar to 
other workers? Where could improved supported employment 
program practices enhance job stability and retention? 
What are the external barriers that prevent supported 
workers from retaining employment? Where do participants 
go after termination f rom supported employment? 

• How is supported employment interacting with the labor 
m arket — what kinds of jobs are being accessed by 

o 18 
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supported workers? What kinds of labor markets developed 
on be'ialf of supported employment projects are being 
tapped successfully? Unsuccessfully? Is supported 
employment filling existing jobs, creating new jobs, or 
both^ and to what extent? 

While not intended to comprise a comprehensive assessment of 
quality, several measurement areas describing basic employment 
characteristics were included by the National Consensus Seminar in the 
core measure set (see Figure 2A). These measurement areas include: 
type of employer, type of job, and reason for terMination from the 
supported employment project, and were viewed by seminary participants 
as descriptive or proxy measures rather than measures of quality or 
program performance accountability. While the basic set of descriptive 
data gathered from these core measures will provide some of the 
information necessary to address the performance questions above, the 
current state of the art in defining and measuring quality of employment 
has certain gaps. For example, measuring the appropriateness of the job 
match is difficult. Current measures reviewed for this study are being 
used during the job placement process itself as a part of job task 
analyses and sk-^ll assessments during service delivery. However, these 
processes do not lend themselves to evaluating the ongoing quality of 
the match or the goodness-of -f it between the worker and the job over 
time. Further development of measures in this area are needed, so they 
can be included in a set of core measures. In addition, any measurement 
of the quality of the work environment itself requires the development 
of new measures. 

Another factor integral to measuring quality of employment involves 
relating the particular supported employment jobs to the opportunities 
in the labor market. While a comprehensive picture of the types of jobs 
performed by supported workers would be useful on national, site and 
local levels for policy makers and project implementors alike, caution 
must be used in com paring the range of jobs developed by individual 
projects given the wide variation in local settings. These measures 



EKLC 



19 



16 



Figure 2A 

QUALITY OF KKFLOYMKNT 
(Employment Characteristics) 
Core ConFensus Data Items and Reconended Measures 



Consensus Data Itez: 


Rec cxsroended Core Meacurec 


• 


Type of employer 


• 


# and X of 


participants employed by different types of 








empl oy ers 






— private nonprofit 










— private for profit - small 










business or large corporatiori 










— government - local, state cr 










— federal 








• 


Type of job (develop standardized 


• 


# and Z of 


participants in different types of jobs 




categories, referencing L.O.T. • 










S.O.C. codes, etc.; 








• 


Who pays vages? 


• 


# and Z of 


participants receiving vages from each source 




— SE program 










— employer 










— other 








• 


Reason for termination by category 


• 


# and Z of 


participants terminating for primary reason by 




(primary and secondary reason) 




category 






— employee performance reasons 










(involuntary departure) 


• 


# and Z of 


participants terminating for secondary reason 




— employer reasons other than 




by category 




employee performance (involun- 










tary departure) 










— employee reasons (voluntary 










departure) 










— provider reasons 










— other 








• 


Participant destination/status after 


• 


# and Z of 


texrminating participants leaving supported 




termination from supported employment 




empl oyment 


for each type of destination 




program, for example: 










— school 










— vait list 










— no service 










~ institution 










— another SE program 










— day activity program 










^- sheltered workshop 










— independent competitive employment 










— retired 










— other 










— unknown 
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Figure 2B 

QUALITY OF BMPUJYMKNT 
Additional or Supplementary Data Iteu and Other Possible Measures 



Additional or Supplementary Data Items 



• ^pe of industry (develop standardized 
categories, referencing D-0-T-, S-0-C-, 
etc. ) 



Other Possible Measures 



• # and X of participants employed by type of indust 



• Reason for termination by specific types 
(more than one may be indicated) 

A. Employee performance reasons 

(involuntary departure), for example: 

— attendance 

— compliance 

— amotional outbursts 

— personal hygiene 

— independence 

— on task (off task) 

— quality of work 

— responding to instructions 

— social skills 

— speed 

— task completion 



B. Employer reasons other than 
emplyee performance (involun- 
tary departure), for example: 

— change in job duties 

— laid-off, facility closed 

— laid-off, cutback in staff 

— laid-off, seasonal 

— replaced by another worker 



C. Employee reasons (voluntary 
departure), for example: 

— quit* due to change in 
relationship status (e.g.» 
married, divorced, etc.) 

— quit, due to pregnancy and/or 
parenting responsibilities 

— quit, for better job 

— quit, didn't like job 

— quit, health reasons 

— leave of absence 
moved from area 



D. Provider reasons, examples to be 
developed 



E. Other reasons, for example: 

— quit, due to parental caregiver 
pressure 

— quit, due to financial aid inter- 
ference 

— quit, due to transportation 
problems 

— deceased 

— other 



• # and X of participants terminating for each type of 
employee reason (involuntary departure) 



# and Z of participants terminating for each type of 
employer reason (involuntary' departure) 



• # and X of participants terminating for employee reasons 
(voluntary departure) 



• # and X of participants departing for provider 



reatsons 



• # and X of participants terminating for other reasons 
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will also be useful descriptors in analyzing reasons for termination and 
participant destination status after program termination. 

Figure 2B provides examples of expanded response categories for 
each of the categories of reasons for termination, vhich have been 
grouped according to employer, employee, or provider reasons. As 
supplementary data* this measure permits recording more than one reason 
f or j ob termination. For example, an employee's behavior which results 
in an involuntary termination m^ be the worker's only w^ of expressing 
dislike for the job. Thus, two explanations, and possibly more, exist 
for that termination. The more detailed response categories would 
permit recording all the reasons for termination that apply to each 
situation. 

It would appear, then, that data on termination reasons may be 
collected at the client level, while other quality of employment 
measures may be appropriata to project level data collection and 
measures to demonstrate overall trendc in job placement and movement 
within the context of local communities and projects. Research efforts 
needed to refine measures of job quality include not only studies of new 
approaches to measuring quality and job match, but also studies to add 
depth to the ongoing descriptive data, such as: (1) job mobility 
comparisons between those in supported employment ai d other workers; 
(2) studies of how supported employment projects are (or are not) 
accessing nontraditional jobs within the labor market; and (3) studies 
of how to use participant self-reports to define and describe the 
quality of employment and changes over time. The current core set of 
measures should be viewed as initial attempts to develop valid 
descriptive measures of the nature and adequacy of jobs. The importance 
of expanding current expertise in measuring employment quality will need 
to be reflected in future research efforts as the supported employment 
program evolves, 

OPPORTONITIES FOR WORKSITE INTEGRATION 

The integration of individuals with severe disabilities into the 
workplace is a key element in the supported employment effort. Worksite 
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integration focuses on providing opportunities for disabled and non- 
disabled co-workers to interact in a variety of settin^.s and situations 
including the immediate work environment, lunchroom, break times, or 
during travel to and from work. 

The results of this study have indicated the importance of going 
beyond measures that simply indicate whether individuals with severe 
disabilities are being placed into jobs where there are nondisabled co- 
workers presant. A more complete definition of integration needs to 
include the existence and extent of meaningful contact between disabled 
and nondisabled coworkers. Evidence indicates that this type of inte- 
raction provides a teaching and reinforcement mechanism for sociaJ. and 
work skill development. It is also an indicator of normalized working 
conditions and the opportunity to establish social relationships between 
people who have formerly been isolated from one another. 

While integration at the worksite is the primary integration 
objective of supported employment programs, study results have also 
stressed the multidimensional nature of integration and the importance 
of integration outside of work. For many participants, success at work 
will be dependent in part on participation and success in integrated 
non-work environments. Persons with disabilities interact with non- 
disabled individuals when buying clothes to wear to work, when buying 
food, and while riding public transportation, as well as during partici- 
pation in recreation/leisure activities. For many disabled individuals, 
community integration is not only an essential pre-requisite to suc- 
cessful integration at the worksite, but is also an outcome of suc- 
cessful participation in supported employment. (See also Quality of 
Life.) 

Community integration is a concern that has been given a great deal 
of attention in recent years by administrators, service providers, 
researchers and policy makers in the fields of independent living, 
developmental services and special education. It is an area that is 
relatively new to vocational rehabilitation and employment policy. 
Perhaps because it is considered by many to be a s ondary outcome in 
the context of employment programs, measuring community integration is 
not yet well— developed or well-defined in the field. Thus, few concrete 
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examples exist of measures of the performance of supported employment 
programs in furthering community integration objectives. Further 
development would be needed to construct simple summary measures that 
could be included in the evaluation of supported employment impacts. It 
appears unlikely that community integration measures would be included 
in ongoing performance assessment^ at least in the initial stages of 
program development. Thus» the proposed core measures listed in 
Figure 3A concentrates on opportunities for integration in the worksite 
only . 

A maj or constraint in measuring integration is the question of how 
to define and collect information on the extent and nature of contacts 
between individual workers. Monitoring the proportion of persons with 
disabilities to nondisabled individuals is somewhat easier and the data 
are :*elatively easy to collect. However^ study results indicate that 
dev" loping indicators of the degree and kinds of interaction would 
result in measures that are more reflective of the accomplishment of 
integration objectives than measures of the numbers of nondisabled 
workers at the job site. 

Some attempts have been made to develop client— level measures of 
integration which include features of both work setting and community 
interaction. While still in the initial stages of implementation and 
reliability testing, such attempts bode well for the development of such 
measures . 

Until such developments occur, or until adaptations are made of 
existing integration measures from other related fields, current 
measurement of integration focuses on measures of proportion, proxy 
measures, and/or qualitative evaluation efforts. As Figure 3A indi- 
cates, the core consensus measures include basic quantitative measures 
of the numbers of supported workers in the group support structure, such 
as work crew or a work station, and the presence or absence of 
nondisabled workers (other than staff) in the immediate work setting. 
In addition, the seminar participants proposed an additional measure 
describing the supervisory structure utilized in the supported employ- 
ment model which includes distinctions between mobile versus stationary 
and individual versus group structures. Each type of supervisory model 



21 



Figure 3A 
WORKSITE INTBGRATION 
Core Consensus Da^a TtemB and Recoosiended Measures 



Consensus Data Items 


Recommended Core Measures 


• TyP^ of supervisory structure: 

— mobile versus stationary 

— individual versus group 

• Presence of nondisabled workers 
(other than staff) in immediate 
work setting during work day: 

— yes 

— no 

group support structure (group 
work station or work crew) 


• # &nd % of supported workers 
in each type of supervisory 
structure 

• % of employed participant ^ 

who work in a setting with 
nondif.abled workers present 

A '2 of T\r r\cy r Sim t^OT"t"n<^TT^flTlt"K Tn 

group support structures of 
eight or less supported 
workers 

• % of program participants in 
group support structures of 
six or less supported workers 

• % of program participants in 
group support structures of 
four or less supported workers 
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has different implicatiop.s for degree of opportunity for integration, 
such as increased opportunities in individual/stationary structures for 
contact and interaction betwee i disabled and nondieabled workers. Thus 
a description of the structure would provide valuable insights about the 
possibilities for worksite integration. This is true, also, in the case 
of measuring the number of supported workers in the group support 
structure — another core consensus measure. All of those measures, 
however, were viewed by seminar participants as proxy measures until 
future research efforts can address these me>«isurement issues further. 

Another proxy measure was developed during the Consensus Seminar 
and was included as a supplementary measure (see Figure 3B) . 
Tracking vhether contacts are required between the jsupported worker and 
supezrvxBorB, covorkers and cuBtomers was viewed as one way to begin to 
study interactions, although this still does not answer the question of 
how to measure the extent and nature of those contacts. 

Determining the proposed supplemental and core data items measuring 
integration raised a number of other issues or constraints. The 
questions were raised: Is integration a practice or an outcome? A 
guideline or a measure? A part of quality of life within aixd outside of 
the workplace? Although no conclusions were reached, the group was able 
to determine that the current sets of proxy integration measures would 
be most appropriately collected at the project level, as they are not 
meant to be measures of individual participant success or failure. 
However, as client level measures of social interaction begin to emerge 
as supported employment programs mature, further discussion is needed to 
develop a common understanding of the role of integration measures in 
measuring program performance. 

ONGOING SUPPORT 

The provision of ongoing support is the "mainstay" of the supported 
employment concept and that aspect which most distinguishes supported 
employment from other employment out comes. The primary obj ective of 
ongoing support is to enable an individual with severe disabilities to 




Figure 3B 

WORKSITE INTEGRATION 
Additional or SupplenentaxT Data izeaxs and Other Pofssible Measures 



Additional or Supplementary 
Data Items 


Other Possible Measures 


• % of disabled workers in work 
setting 

— total # of workers in work 
setting 

— total # of workers with dis- 

bilities in work setting 

• During the course of the work 
day, is the supported worker 
required to interact with the 
following : 

— supervisors, 

— coworkers* 

— customers? 


• # and Z of supported workers 
in work settings where less 
than 10% of the workers have 
disab ilities 

o # and % of supported workers 
in work settings where 11-50% 
of the workers have disabili- 
ties 

• if and % of supported workers 
in work settings where nore 
than 50% of the workers have 
disabilities 

• % of supported workers 
required to interact with: 

supervi sors 

coworkers 

customers 
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sustain employment at a level that he or she would be otherwise unable 
to achieve. 

The most common concept of support is the presence of a supported 
employment program staff person to provide on*^the-job training and 
assistance. While concepts about the nature of these responsibilities 
vary, most models of supported employment include on-the— job support in 
three major areas: training, supervision, and behavior management. In 
addition to specific work-related support, the concept of supported 
employment may also include support that is indirectly related to the 
employment objectives such as transportation assistance, "peer support," 
and services to employers. Other types of support indirectly related to 
employment are those related to life activities outside of work such as 
assistance with independent living and social skills development. There 
is substantial difference of opinion about the role of supported employ- 
ment programs in addressing needs outside of the workplace* 

Diversity also exists in the intensity and duration of the support 
provided to participants. While the basic premise of ongoing support is 
that it is provided on a potentially permanent basis, some programs 
operate models in distinct phases in which on-the-job training is faded 
out over time and replaced with follow-along services that focus mor^ on 
independent livings social skills, and general problem solving than on 
the work itself. In other programs, the structure of the job-related 
support is ongoing by its very nature, as in group work station or 
mobile crew models. 

Although the concept of ongoing support is key to understanding how 
supported employment differs from other forms of employment, discussions 
about the implementation of performance measures related to ongoing 
support have been inconclusive. It was suggested that measures of 
ongoing support should be used es descriptive measures of different 
supported employment models, rather than as noi native measures , since it 
is not the case that providing one level or type of support is better or 
worse than another. The ideal is to provide the necessary support to 
assist each participant to work as independently as possible, which will 
be a different mix for each participant and may vary f or a given parti- 
cipant over time. 
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As described in Figure 4A, three data items were identified as 
core measures for the domain of ongoing Gupport. These include: 

• a description of the tjrpee of support which the project 
generally provides to its participants (e^g,, assistance 
in transportation to and from work or self-care, training 
in job skills, social interaction/behavior, or community 
living skills, or other support); 

© i:?hether the prcjact generally provides support *it the 
work place, outside the vork place, or bothi and 

• a description of the general f ramework vithin which 
support is provided by the project (i.e,, in an 
individual placement setting, in a group work station, in 
a mobile work crew, or in a small business run by or for 
employees with disabilities). 

These data items can be used to develop project-level measures that 
describe the features of particular supported employment projects. 
Discussions with representatives of a variety of projects revealed that 
not all proj ects document variations in the types or amounts of support 
received by individual participants for use in a client-level measure. 
However, it would be useful in describing each supported employment 
project to know the approximate proportion of participants who received 
each type of support during a given reporting period, in addition to 
knowing whether the project has the ability to provide support of each 
kind. 

Figure 4B describes several additional data items and measures 
that could be used to describe the delivery of ongoing support in more 
detail. These data items m^ be perceived as relevant to some supported 
employment models and not others, and may be available at some projects 
and not others. The data items include: 

• what funding sourceCs) are used to provide ongoing 
support; 

• vhat organizations or individuals actually provide 
ongoing support (e.g., supported employment staff versus 
• cp2 
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Figure AA 

ONGOING SUPPORT 
Core Consenstis Data Items and Recoonaended Measures 



Consensus Data Items 


Recommended Core Measures 


m ly pe of support prcrv ided : 

— transportation 

— self care 

— job skills 

— social interaction/behavior 

— community living skills 

— other 

• Whether support is provided: 

— at the work place 

— outsxde the work place 

— both 

• Type of support structure : 

— individual setting 

— group work station 

— mobile crew 

— small business run by 
or for employees with 
disab ilities 

— other 


• type of support prorvided by 
project during reporting 
period 

• and % of participants in 
different types of support 
structures 

• approximate proportion of 
supported employment par- 
ticipants receiving each 
type of support during 
reporting period 
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Figure 4B 
ONGOING SUPPORT 

Additional or Supplementary Data Items and Other Possible Keasures 



Additional or Supplementary 
Data Items 


Other Possible Measures 


• 


Funding source (s) for publicly- 
funded ongoing support (i. e , , 
agency and budget line) 


• 


Funding sources utilized 


• 


Provider (s) of publicly-funded 
ongoing support (i«e« p supported 
employment staff versus compensa- 
tion to employer^ coworkerfc or 
local service agency) 


• 


Providers utilized 


• 


Total monthly hours of support 
per participant 




Mean hours of ongoing 
support per participant 
during reporting period 


o 


Total monthly hours of support 
across all participants for each 
type of support 




— by type of support 

— in total 


• 


Total monthly costs of support 
per participant 






• 
• 


Total monthly costs of support 
across all participants for each 
type of support 

Total earnings by supported 
employment participants ^ 
during the reporting period 


• 


Mean and distribution of 
ratio of cost of ongoing 
support to earnings gener- 
ated per participant during 
reporting period 



See Data Items under Employment Outcomes 
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employers or coworkers^ versus another local service 
agency) ; 

e hov Many hours of oupport are provided each months in 

total, and by type of support; and 
o what the aonthly coat of providing ongoing support is, in 

total and by type of support. 

These data items can be used to compute descriptive statistics 
including the mean Lours and dollars of ongoing support provided each 
Month per participant. By using data on mean monthly earnings 
(described under Employment Measures) an additional summary measure can 
be computed of the Mean Monthly cost of providing ongoing support 
compared to the Mean monthly earnings generated per participant during 
the reporting periodL 

In summary, while the descriptions of ongoing support are not 
intended, by themselves, to provide an assessment of program success, 
they are critical for understanding variations in the service strategies 
and resulting costs of different models of supported employment. The 
institutional linkages and funding flows by which continuity of 
services/ongoing support is maintained over time is another key 
imp-1 ement at ion issue with wide variation- A careful study of this 
aspect of supported employment may yield ideas about "best practices" 
that will prove valuable in replication efforts. 

QUALITY OF LIFE 

Improved quality of life for individuals with severe disabilities 
has consistently been raised as a desired outcome of supported employ- 
ment. How improved quality of life is defined and measured varies from 
person to person, but in general the concept is viewed as increasing 
those activities which bring pleasure and self-worth to the severely 
disabled individual over his/her lifetime. The activity of work by its 
very nature has intrinsic values which can bring a sense cf satisfaction 
and fulfillment, and can increase access to and privileges from society 
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which are most commonly associated with work (i.e., making friends, 
sharing resources, and expanding social skills and environments). 

Parallel to the personal intrinsic gains obtained by an individual 
with severe disabilities when he/she has the opportunity to work are the 
positive changes that can occur for the family or primary caregiver of 
the employee in supported employment. Reduced family stress, and 
freeing up time for family members are additional quality of life out- 
comes. Quality of life can also improve as a result of the expanded 
activities an individual can participate in or the additional items 
he/ she can obtain because of his/her increased purchasing power and/ or 
greater access to choices. In addition, improved qual ity of life can 
result from a combination of more self-confidence, self -direction and 
increased exposure to a greater array of options in the world. 

It is only reasonable that quality of life issues focus on the 
individual and how the individual makes choices to obtain greater satis^ 
faction. Yet, in the area of supported employment, the project and the 
overall system provide the structure which determines whether the 
individual with severe disabilities understands choice, has had practice 
making choices, and knows how to choose for the purpose of satisfying 
his/her needs. The concept of individual choices is considered to be an 
important aspect of quality of life. 

Through discussions with respondents and reviews of the current 
available literature on measure development, the BPA study team identi- 
fied the following aspects of quality of life as areas for potential 
measurement : 

• type of living arrangement at program entry and ov er 



• use of aiainBtreaaL transportation at program entry and 
over time; 

• degree of community integration and participation; 

• changes in Quality of Lif e Indicators, e. g., health 
status, self-direction and opportunity for choice, 
attitudes of family, participants and/ or caregivers, 
self— esteem, skills levels, etc. ; and 



time; 
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• expendable income of participants^ at program entiy and 
over time. 

During the National Consensus Seminar participants indicated that 
the above areas did not adequately define the concept of quali ty » 
suggesting that these aspects actually described "lifestyle 
characteristics" rather than "quality of life." Some participants 
raised the issue that using the term 'Quality" implied a value judgment^ 
whereas looking at changes in characteristics over time v/ould not 
involve defining the standards for quality. Participant self-report was 
Suggested as a way to capture this information without arbitrary values 
being placed on changes by non-participants. No specific data items» 
however, received group approval for inclusion in the core data set, 
although the domain itself was considered important to continue to 
pursue through research and special studies. 

Several constraints emerge when considering the "how-to's" of 
measuring quality of life. At this point in time, a refined and 
reliable quality of life questionnaire and response index, which could 
be used across supported employment participants, family members and 
caregivers, does not exist. This kind of instrument would enable base- 
line client-level data to be collected at program entry and then 
compared over time with results from administering the same 
questionnaire at regular intervals. Some work is currently being done 
in this area, and there appears to be growing interest in further 
research ef forts. There is a concern, however, that a standardized 
questionnaire may not be feasible, given differences in local 
communities and services. Limitations in what are considered 
"acceptable" evaluation methods, i.e., use of participant observation 
techniques or self report, may also curtail the development and use of 
innovative creative approaches to quality of life measurement, to the 
extent that policy makers and funding agencies may not consider theci to 
be valid. For non-verbal participants, however, using alternative forms 
of self-report may be the most effective method. 

The zieasurement of quality of life changes also involves the issue 
of whether defining or standardizing terms, such as choice and autonomy. 
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is possible or even desirable. While most seminar participants agreed 
that these concepts were integral parts of quality of life, the question 
of definition and ultimately, of feasibility of measurement arose. When 
considering choice, there appear to be three major dimensions: (1) the 
opportunity for choice, i.e., whether an individual is living, working 
and plying in an environment where the possibility of self-direction 
exists; (2) the extent of choice, i.e., how m uch possibility for 
decision-making exists; and (3) the kind of choice, i.e., the range of 
areas available within which to make decisions. 

While no consensus was reached on core measurers, as shown in 
Figure 5, several measures were included as supplemental proxies for 
changes in "lifestyle charactersi tics." These include type of living 
arrangement, use of maxnstreaa transportation, and a self-report measure 
of degree of independence in living situation. It was noted that 
measuring type of living arrangement should include lateral movement 
within the category as potentially more or less independent than the 
participant's previous situation. Measuring use of mainstream transpor- 
tation would also be a supplementary measure used at project discretion, 
given that transportation and community mobility is locality specific. 
The self-report measure was added during the seminar to allow for 
participant definition of independence. 

Assessing quality of life changes for persons with disabilities 
remains a critica? issue for development and refinement of current 
measures. The intent of supported employment is to positively influence 
quality of life through meaningful paid work in integrated settings. 
Whether or not it ever becomes the responsibility of programs to track 
these influences remains to be seen, but as an avenue for encouraging 
the active self-assessment of the impact of supported employment on 
participants' lives, quality of life measurement represents an essential 
part of evaluation efforts. 

PARTICIPANT CHARACTERISTICS 

The target population for supported employment programs is that 
group of individuals traditionally excluded from vocational 
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Figure 5 

QUALITY OF LIFE 
(Lifestyle Characteristics) 
Supplementary Data Items and Other Possible Heasures 



Supplementary 






Data Items 


Other Possible Measures 


• 


Type of living arrangement at 


o 


# and % of participants in 




program entry nnd over time » 




each type of living arrange- 




X or ejLamp^ e • 




ment at the end of each 








reporting period 




— with parents or relatives 








in group nome or resiaentiaj. 








f* "5 T "i trv 


w 


V anu ^ or par uxcipanus 




— in semi— i n Hppenden t 1 iving 




changing t o more independent 




5 i t ua t i 




settings during reporting 








T^OT""l oH 1 C ^1 T" ^ OT"T" I 
















Jh ar^H ^ t^fiT"'t""5r»"5T^flTTt"W T" 
IT oilu A> yJi. ux^x^oiiuo xc 




independence within living 




porting increased indepen- 




setting 




dence within living situation 


• 


Use of mainstream transporta- 


• 


if and % utilizing mainstream 




tion at program entry and 




transportation with assistance 




over time: 








— does not utilize mainstream 


• 


# and % utilizing mainstream 




transportation 




transportation independently 




— utilizes mainstream trans- 








portation with assistance/ 








prompts 


• 


# and % of participants 




— utilizes mainstream trans- 




increasing use of mainstream 




portation independently 




transportation during 








reporting period 
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rehabilitation services due to the severity of their disabilities. 
Supported emplpyinent is designed as an alternative to day activity and 
habilitation programs. Most proponents of supported employment 
subscribe to the concept of the "zero reject model", which suggests that 
no individuals would be turned away from supported employment programs 
solely on the basis of limited abilities. This does not imply "total 
inclusion" or that all individuals must work. However, the "zero 
reject" concept does imply that individuals will not be denied the 
opportunity to do productive work just because their capacity for work 
is less than the standards set for individuals without disabilities. 

Because of the interest in using supported employment to reach out 
to individuals excluded from work opportunities in the past, and because 
the supported employment approach may be appropriate for a wide range of 
individuals with physical, mental, or emotional disabilities, it will be 
important for programs to collect data which capture the range of 
participant characteristics. Three major questions arise when assessing 
the performance of supported employment programs: 

(1) Who is being served : 

(2) Who is being successfully served? 

(3) How do out comes and service approaches vary for 
different types of participants? 

In the past, the client characteristics recorded on supported 
employment intake/assessment records at the project level have usually 
included variables that are viewed as relevant by program operators in 
the development of a service plan to meet each individual's needs, as 
well as data that are readily available at the time a participant enters 
the program. Given the potential interest in using participant data 
(1) as an indication of whether appropriate individuals are being 
served, and (2) as an interpretive tool in assessing and comparing 
reported project out comes> a broader range of participant descriptors 
needs to he developed. The candidate measures can be divided into 
several categories : 

• demographic da^a that are easy to measure and record 
(e.g», age, sex, ethnicity); 
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• descriptors of the participant's disabling condition and 
functioning levels (which are less easy to measure and 
categorize) ; 

• the individual's situation inaediately prior to entering 
the supported employment project; and 

© previous work history and previous service historj. 

Participant Demographics 

Scpported employment programs are all currently collecting the 
basic demographic information on participants served. An issue which 
needs additional discussion in efforts to further a consistent reporting 
system is **who is considered a participant in supported employment** — 
all individuals referred to the system^ all individuals for whom an 
assessment is completed, all individuals who enter a training phase of 
services* or all individuals who are currently being compensated for 
work under the program? While variations in program models may make i*; 
impossible to arrive at complete standardization on this issue, more 
discussions are necessary to promote the greatest possible consistency. 

Descriptions of Disabling Condition and Functioning Level 

There are two distinct reasons for collecting data in this 
category. One reason is descriptive ; to capture the variations in 
types of disabilities and functioning levels exhibited by project par- 
ticipants across projects and across states. Another reason is 
evaluative ; to assess the •'severity" or potential difficulty of the 
participants selected for services, both to monitor whether the group 
selected for services matches the stated program goals, and to assess 
the level of project outcomes, taking into account the chc^acteristics 
of the clients served. For the second purpose, it may be useful to 
construct a "client difficulty index" based on the answers to the indi~ 
vidual data items listed here. However, until projects finalize the 
types of disability categories and groupings they anticipate including 
in their participant population, it may be premature to suggest how to 
construct such a measure. In addition to referring to the variables 
describing disabling condition and functioning level, a client diffi- 
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culty index would probably also be based on nome of the variables in the 
last category — previous work and service history. 

Situation at Program Entry 

The three variables that the study team clustered into this cate- 
gory included (1) current living arrangenent6» (2) public assistance 
status at enrollsient, and (3) service setting ismediately prior to entry 

into the supported employment program* These are each f^ata items that 
should be relatively easy to collect, since they are based on partici- 
pant status at a given point in time (i.e., project entry) rather than 
reconstructions of experience over a longer preprogram period. The 
description of living arrangement and public assistance status are 
important not only as descriptors *"o capture the range of variation in 
participants across projects and states, but are also important baseline 
measures for use in tracking individual changes brought about as a 
result of project participation. 

Previous Work and Service History 

The variables clustered in this category can be used in three 
different ways: (1) to identify participant characteristics that may 
influence the level of outcomes a participant is likely to achieve 
(x«e., capturing some aspects of client difficulty); (2) to record more 
complete descriptions of an individual's preprogram experience that can 
be compared to the same individual's experiences after entering 
supported employment in order to construct change measures (e.g., change 
in employment intensity, change in earnings, and change in work 
setting) ; and (3) to identify more completely what groups of people are 
being reached by the supported employment demonstration (and what groups 
are not being reached) in order to assess the extent of system trans- 
f ormation that is occurring. 

According to the individuals involved in the national consensus 
process, those participant characteristics considered important for 
inclusion in the core data set (see Figure 6A) included measures from 
each of the four categories of demographics, disability informtion, 
situation at entry and prior setting. The core measures were generally 
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Figure 6A 

PARTICIPANT CHARACTERISTir'45 
Core Consensus Data Items and Recoomended Measures 



ConsensuB Data Items 



Recommended Core Measures 



Age (date of birth, with aggre- 
gate data reported in cate~ 
gories Under IS, 18-21, 22-30, 
31-40, Al-64, 65+ older) 



• mean age 

• % by age categories 



Sex 



• % by sex 



• Primary disability (by maj or 
category) 



# and % of participants with 
each primary disability type 



IQ 



• mean IQ level 



• Public assistance status at time 
of program entry 

— SSI 

— SSDI, etc. 



# and % of participants re- 
ceiving public assistance 
by category- 



Amount of cash grant (check state 
of art for feasibility and defini- 
tion of what to include) 



mean monthly grant 



Service setting immediately prior 

to program entry, such as: 

— special education — segregated 

— special education - integrated 

— adul t ed/ c ommuni ty c ollege 

— sheltered workshop 

— work activity center 
- — day activity center 

— institution 

— other SE program 

— no day program 

— employed 

— other 



% of participants from 
immediately prior service 
settings by category 

% of participants from 
day activity programs 



# of participants ever in each 
prior setting (if research shows 
this has predictive value) 



# and % of participants 
ever in each prior 
service setting 
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felt to be deecriptxve in nature rather than measures to which programs 
should be held accountable, though consensus was not reached or this 
issue. 

There was much discussion about participant characteristics and 
concern that they not be used inappropriately in conjunction with 
outcome data to target the program towards certain types of individuals. 
Historically, under a medical modelg characteristics and especially 
functional assessments have three uses, descriptive, prescriptive, and 
predictive, with the predictive end considered to be of the "highest 
good." The goal of this information for the supported employment 
program, however, is not one of attempting to predict success (because 
the program is designed to succeed for those who would otherwise fail), 
but rather one of providing appropriate accommodations to individual 
needs and providing opportunity for choice. Thus, evaluation using 
participant data would look at what works for different individuals, not 
whether it works. It can also be used to predict costs. 

The following data items were considered core by some seminar 
particpants and supplementary by others. These issues were not resolved 
in the large group consensus process and are included as supplementary 
items in Figure 6B : 

• ethnicity; 

• primary language; 

• presence of disability by category (more than one may be 
indicated) ; 

• severity of disability (other than mental retardation), 
with the suggestion of developing a summary score or 
index for severity of disability across all types of 
disab ilities ; 

• living situation (at time of program entry) ; and 

• couununication skills. 

SYSTEMS CHANGE 

According to those who are active in supported employment efforts, 
project-level endeavors need to be accompanied by system-level 
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Figure 6B 
PARTICIPANT CHAaACTEBaSTICS 
AdditiooAl or SuppXe»«nt«ry Data Xtema and Other Possible Heasures 



Additional or Supplementary 
Data Items 


Other Possible Measures 


• Ethnicity 

— White 

— Black 

— Hispanic 

— Asian 

— Native american 

— Othfir 


• % by ethnicity categories 


• Prij&ary language 

o Presence of disability by category 
(more than one category may be 
Indicated) 


• X vith English as a second language 

• and X of participants vith each disability type 


• Detailed diagnosis 

• Expreaaive coomunication aV.ills 

— unclear speech 

— clear speech 

— vocalization only 

— uaes aids or sign language 


• X %''ith secondary disability 

• list and number by diagnosis 


• Receptive communication skills 

— understands gestures 

— understands words and phrases 

— understands coxrversations 


• X by category of expressive communication skills 


• Aaaessment of severity of mental 
retardation 

— profound 

— aevere 

— moderate 

— mild 

— oot applicable 


• X by category of receptive communications skills 


• Assessment of severity of other types of 
disabilities (measures to be developed) 


• X by category of severity of mental retardation 

• X by category of severity of other disability 
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procedural and policy changes to enable program and individual level 
outcomes to occur within a statewide framework of coordination, support, 
and cooperation. The ability of existing systems to facilitate the 
transition to supported employment from the current configuration of 
vocational, pre-voca t ional, and day activity service programs will 
depend on the ability to confront and redefine policies, procedures, 
funding streams^ and program structure^ Without these changes on all 
system levels, supported employment may flounder and fail to develop a 
c .^rehensive resource allocation and coordination strategy. 

There is some disagreement about whether system transformation is a 
secondary outcome that results from effectively addressing participants' 
supported employment needs, an interim set of procedural steps necessary 
to accomplish supported employment outcomes at more than a token level, 
or whether it is a primary objective in and of itself. Some program 
administrators, operators^ and state policy makers have indicated that 
their primary concern is for participant outcomes, and that if these 
outcomes are pursued, systems change will occur to the extent that it is 
required. Others suggest that measurement of participant outcomes alone 
is insufficient for assessing performance, noting that it may be 
possible to achieve supported employment outcomes for a limited number 
of individuals without bringing about the broader structural changes. 
These changes in the service system would be needed to ensure the 
availability of supported employment opportunities to the larger 
population that could benefit from access to these opportunities. 

If a performance measurement system is to include an assessment of 
the extent to which system transformation has actually occurred, then 
system change measures need to be developed and implemented. A review 
of existing measurement systems reveals little in the way of existing 
measures of system change currently in use, presumably because most of 
the supported employment activity occurring to date has been in the form 
of specific projects rather than as a system transformation activity. 
However, a review of existing literature and programs has been helpful 
in identifying the areas in which measurement is needed. 

These measurement areas include (1) the extent of local systems 
changes, such as changes in local case management and referral 
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mechanisms: (2) the influence of the supported esiplcyment initiative on 
the fitate service delivery systeau including achievement of state objec- 
tives, and changes in service delivery mechanisms; (3) the volume of 
supported employment outcomes; and (4) the fimding flovs or amounts of 
funding by source and the relations between the funding flow and the 
participant flow into supported employment. 

Those cysteras change outcomes considered important to include in 
the core data set included the more concrete measures such as the 
proportion of individuals being served and the flow of funding by 
different agencies. The systems change measures included are 
constructed using data items from other clusters, as well as using an 
additional data item asking for amount of supported employment funding 
by source (see Figure 7A). 

Rather than focusing on individual core data items for this domain, 
the participants in the National Consensus Seminar chose emphasized 
sximmary performance measures of systems change. It was noted that when 
considering systems change, one must answer the question: 
What are we trying to change ? The suggested answers to this question 
included : 

(1) increasing opportunities for individuals with severe 
disabilities; 

(2) reducing the numbers of individuals in day activity programs; 
and 

(3) shifting funding patterns from traditional day programs into 
supported employment . 

Thus, the three measures in the core set provide the mechanisms for 
assessing the extent to which these changes have occurred. 

Measures of achievement of state objectives, shifts in local 
service delivery systems and data items requiring information data from 
outside of the supported employment system (e.g., number of participants 
in day activity programs) were included in the supplementary set of 
measures (Figure 7B), These supplementary measures, then, include 
measures and data items that are particular to states and local 
communities, requiring development and collection at state and/or local 
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Figure 7A 
SYSTEMS CHANGE 
Core Consensus Data Items and Recoaunended Measures 



Consensus Data Items 


Recommended Core Measures 


[The first two recommended core 
measures use data items from the 
Employment Outcomes section, in 
conjunction with data from out- 
side of the supported employment 
system on H'lmber of total par- 
ticipants in other programs and 
waiting lists] 

• Amount of funding for supported 
employment programming by source 


• Volume of supported employ- 
ment outcomes over time 

• Proportion of eligible target 
population being served 

• Ratio of supported employment 
participants to total publicly- 
funded day program participants 
(including day activity, work 
activity, tnd sheltered work- 
shop programs) 

• Amount and proportion of 
funding for supported employ- 
ment programming by different 
funding sources over time 
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STSTKHS CHANGE 

Additional or Supplementary Data Xtema and Other Possible Keasuree 



Additional or Supplementary Data Items 


Other Possible Measures 


• 


State objectives for systems change 


• 


Evidence of state objectives of systems change in legis- 








lation, regulation* agency roles and relationships » 








funding mechanisms, etc. 




Achievement ol stat^ objecciv*iis o£ 


• 


Evidence cx: ochievement of systems change objectives 




aystems change 




• 


Cbanges in local service delivery 


• 


Evidence of change over time in each of the components 




system, a.g.. 




of the local service delivery systems 




— referral structures 








— interagency cooperation 








— ccntralixcd coordination 








rasponsibil ity 








— funding sources for ongoing 








Support 








— isxvolvenent of parents* consumers. 








and caregivers 








— case management procedures 






• 


Cbanges in state service delivery 


• 


Evidence of change over time in each of the components of 




Bcchanisms such as: 




the state service delivery system 




case management procedures 
















— ability to provide ongoing funding 








— referral mechanisms 








— documentation and monitoring 








— interagency coordination mechanisms 






• 


# of participants in supported employ- 


• 


Aatio of supported employment participants to partici- 




ment (uses data from client character- 




I>ant6 in each of the follcwing progrcms (who have not 




istics) 




achieved supported employment) : 








— day activity 


• 


# of participants in day activity pro- 




— work activity 




grams, work activity programs, and 




— sheltered workshops 




shelf^sred workshop progams (uses data 








from outside of the SE system) 


• 


Changes in above ratios over time 


• 


Immediately prior setting (see partici- 


• 


Changes in proportion of participants from different 




pant characteristics data) 




immediately prior settings over time 


• 


Amount of system funding for each 


• 


Proportion of funding for each component of supported 




component of supported employment (e.g.. 




employment which is paid for by the different funding 




ongoing support, administration, staff 




sources over time 




training) by source 
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discretion. Constructing measures using data collected fx^om outside of 
the supported employment system would require coordination between 
different agencies for some states — a measure of interagency 
cooperation itself. 

System change measures reflect the information that a state or 
local system will choose to collect for monitoring its own transfor- 
mation. These data would likely be collected annually and would provide 
both a statistical and descriptive record of what has occurred since the 
program's inception. This information will become increasingly valuable 
as the program's longevity increases. Not only will system change 
measures document the system's commitment to and follow-through on long- 
term plans for increasing opportunities for persons with severe 
disabilities^ these measures will also prove useful for looking across 
states at a national picture of the program's accomplishments. 

SOPPORTED EMFLOTMENT COSTS 

The implementation of any new program or service approach raises 
questions about how much the program costs and how ite costs compare to 
those of other programs. This is especially a concern when» as in the 
case of the supported employment initiative^ the goal is one of system 
transformation* not merely one of starting new programs. Thus» 
observers not only want to know how much the program costs* and how 
these costs compare to other programs* but also want to explore the 
relationship between the costs and the outcomes achieved through the 
program. 

Program costs are a critical data element in order to (1) compute 
cost effectiveness measures for the delivery of supported employment 
services; (2) track how the public investment in supported employment 
programming is changing over time; and (3) assesG how the public 
and societal financial investment in supported employment compares to 
the taxp^^er financial benefits generated by the program. 

Attention to the performance of supported employment on cost 
measures implies a series of comparisons. One such comparison is 
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between supported employment and other day programs, to address 
questions such as the following: 

• How do supported employment costs per participant compare 
to the costs of alternative day programs, such as day 
activity centers, work activity centers, and sheltered 
workshops? 

• How do the financial returns of supported employment (tax 
revenues and reduced transfer payments from the 
tar:payer's perspective, and increased disposable income 
f r om the participant' s perspective) compare to the 
financial returns generated by public investment in 
alternate day programs? 

• Do the cost savings generated justify the public outlay 
for these programs? 

A second set of comparisons is among different supported employment 
projects, and particularly across different models of service delivery 
(e.g., across projects that utilize individualized job settings versus 
mobile work crew model s versus enclaves within industry) to answer 
questions such as the following: 

• How do the costs of supported employment vary f rom 
project to project? 

• Do cost-effectiveness or benefit-cost measurers vary 
according to the model of supported employment that is 
being used? 

It is important to note here the need for caution in conducting 
comparisons such as these. Care will need to be taken to ensure that 
program context is taken into account as well as the characteristics of 
program participants. There are many variables that can influence costs 
besides the operation of the programs themselves. 

In negotiating consensus about the cost data items that should be 
considered core or universal data elements, several constraints were 
identified. First, not all supported employment projects monitor costs 
in a way that would permit them to identify the actual costs associated 
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with serving a given individual. (Group-oriented models, for example, 
are less likely than individual placement models to keep track of the 
specific levels of support provided to individual participants). Thus, 
the recommended core measures focus on collecting aggregate project data 
on program costs and computing mean costs per participant, rather than 
on recording costs for each participant. Second, a number of projects 
^'ould find it difficult or impossible to distinguish the costs 
associated with different program activities (e.g., ongoing support 
versus supervision, since in many projects these activities are closely 
intertwined or conceptually merged). Thus, the recommended core data 
items ask only for total program expenditures. 

Figure 8A summarizes the data items that are recommended for 
universal data collection, and the recommended performance measures that 
are based on these data items. The core data elements document 
supported emplcayment funding sources and expenditure levels for a given 
reporting period (such as an annual period), and document any 
restrictions on the services, participants or time frames imposed by the 
funding sources. Public funding as well as privately-generr ced revenue 
are included. 

The recommended performance measures based on these data items 
summarize total project expenditures from public and other revenues, and 
compute nean total annual cost per enrollee» nean onmial public cost per 
enrollee, and the mean public cost of supporting one participant in 
supported employment for one montlu A final recommended performance 
measure utilizes aggregate statistics on mean monthly participant 
earnings (described under Em ploy m ent Measures ) to compute the mean 
public cost per dollar of participant earnings. 

Figure 8B summarizes a series of supplementary cost analyses that 
are needed to answer performance questions about the costs, benefits, 
and relative cost effectiveness of supported employment compared to 
other programs. Each of the items listed in Figure 8B really 
represents an extended set of data elements and a series of 
computations. 

The first supplementary analysis involves comparing the total costs 
and mean monthly costs computed for supported employment projects to 
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Figure 8A 

SUPPORTED EMFLOTKKKT OOSTS 
Core CoaAensus Data Iteu and Recoeaended Measures 



Core ConBensus Data Items 



Amount of Public Funding by Source 

• Funding agency 

• Total amount by funding source 

• ConRtrnints by funding source 

~ Funding constraints about time 
frame (s) 

— Funding constraints about service 

— Funding constraints about eligible 
or target populations 



Recommended Measures 



# and type of public funding sources 
Amount of public funds from each source 
Total public erpsnditurcs during reporting pc 
X of public funds from each source 
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Amount of Other Funding by Source 

• Nonpublic aources of funding 

— Foundation grants 

— Community fundraising 

— User fees 

— Revenue generated by sale of 
products 

• Total amount by funding source 

• Constraints by funding source 

Funding constraints about time 
f rame (s) 

Funding constraints about services 

— Funding i:on8traints about eligible 
or target populations 



• # and type of other revenue sources 

• Amount of funds from each source 

• Total expenditures of other-than-public fundr during 
reporting period 

• Z of other rcfvenues from each source 



Total Program Costs 

• Total public costs for supported 

employment during (annual) reporting 
periods 



• Total expenditures (public and other) 

• Z of total expenditures that are public 

• Mean total cost per enrollee 

• Mean public cost per enrollee 

• Mean public cost per person month of supported 
employment 

• Mean public cost per dollar of participant earnings 
during reporting period 



Total Public and Private Costs During 
(Annual) Reporting Period 

(For revenue producing projects and 
projects with nonpublic sources of 
support for total expenditures including 
revenue generated by project or other 
non-public funds) 

• Total enrollees served during reporting 
period* 

• Total individual person-months of 
supported employment accumulated 
during the reporting period* 

• Total earnings by supported employment 
participants during the reporting period 
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Figure 8B 
SUPPORTED KMPLOYHEKT COSTS 



Additional or Supplmentary Data Itou and Other Poaeible Measuret* 



Additional or Supplementary 
Data Items 


Giner Possible Heasuies 


Funding Details 

• Amount of funding for each type of service 
(if Applicable) 

• Amount of funding for each type of partici- 
pant (if applicable) 

Relative Costa of Alternative, Adjunct or 
Complementary Programs Utilized by Supported 
Employment Participants 

• Total monthly cost of other programs during 
reporting period 

• Total monthly cost per enrollee in paid work 
through other programs 

• Total monthly cost per participant dollar earned 
through other programs 

• Cost of participation in day services as an 
adjunct to supported employment 

Taxpayer Impacts: Computation of Taxpayer Benefit- 


• Services provided under each funding source 

© i and Z of participants served under each funding 
source 

• Generation of summary cost data; comparison of cost 
measures (see recommended measures) between 
supported employment and other day programs such as 
sheltered work activity programs, day activity 
programs 

• Generation of summary cost data; comparison of 
taxpayer costs and benefits between supported 
employment and other programs 

• Generation of sunmary cost data; comparison of 
mean participant expendable income between 
Supported employment and other programs 

• Generation of summary cost data; mean net costs 
(benefits) of employing supported workers (in 
comparison to labor industry standardt; or 
co-workers in similar jobs) 

• Percentage of budget allocated to each type of 
activity 

• Shifts in budget percentages from one reporting 
period to another 
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Cost for Supported Employment ProgrAm^ 


Compared to Other Prof^rams 

• Mean monthly transfer payments 

• Mean monthly cost of social services (including 
cost of day programs) 

• Mean value of tax revenues generated 
Participant Impacts: Computation of Participant 


Benefit-Cost for Supported Employment Compared 


to Other Programs 

• Mean monthly earnings, net witheld taxes 
a Mean reimbursed work expenses 

• Mean monthly transfer payments 

Employer Impacts: Costs and Benefits of Employing 


Supported Versus-Non-Supported Workers 


• Provision of training and supervision 

• Job accommodation costs 

• Productivity levels 

• Absenteeism 

• Job turnover rates 

• Wages paid 

• Fringe benefits provided 
Costs by Type of Project Activity 

• Cost or percentage of total budget 
allocated to: 

— general administration 
outreach, job development 

— participant intake, assessment, case 
fflanagement 

— direct participant supervision, 
^'~»ining, ongoing support 

P n f /^rchase of outside services for participant 
^enue producing activities (including cost 
hiiiimiBTir[Ti™i wages paid for work performed) 
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data on the costs associated with other day programs. For the purposes 
of comparison with other programs, the full costs of supported 
employment also include the cost of participation in other day services 
as an adjunct to supported employment. In addition to comparing total 
program costs, it will also be important to do comparisons using 
standardized units such as "cost per participant hour" and "cost per 
participant dollar earneda" The generation of data on costs of other 
programs would have to be undertaken at the state level or local system 
level, since these data are not available to local supported employment 
projects. 

The second supplementaiy analysis involves comparing the financial 
costs and benefits of supported employment from the perspective of the 
taxpayer. To generate these measures, it will be necessary to collect 
data on: 

• supported employtnent program costs ; and 

• other changes in public costs resulting from participa- 
tion in supported employment, such as: 

— changes in transfer payments received by participants 

— changes in participant utilization of other publicly- 
funded community services 

— sa xngs from increased tax Tpvenues paid by 
participants. 

It is likely that collection of these data, even for a limited research 
sample, will be bey ond the capacity of most support ed employment 
projects, unless special research funding and research staff are made 
available. In order to compare the taxpayer benefits and costs from 
supported employment to the other day programs, similar data on costs 
and benefits would have to be generated for these other programs as 
well. 

The third supplementary analysis, involves comparing the costs and 
benefits of supported employment f romthe perspective of society as a 
vhole. To generate these measures it will be necessary to collect data 
on: 
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• total costs of supported employment and other social 
services to participants including non-governmental costs; 
and 

• the value of participants* production or contribution to 
the productivity of society as a vhole — i.e. net 
contrioution to the social product. 

This analysis also provides the specific vehicle for addressing the non- 
economic benefits of the program such as increased community integr tion 
and participation and improved quality of life for participants and 
their families. 

The fourth supplementary analysis involves comparing the financial 
coats and benefits of supported employment from the perspective of the 
progras participant. In computing these measures, it will be necessary 
to collect data on: 

• unreimbursed participant expenses associated with par-- 
ticipating in supported employment; 

• increases in participant take-home earnings; 

• changes in earnings of other household and family members 
due to reduced need for participant care; 

• changes in transfer payments or other financial support 
received by participants; and 

• estimated value of any changes in life quality 
experienced by participants. 

Like the previous taxpayer benefit/cost measures, the collection of data 
for the participant benefit/cost measures is likely to be beyond the 
capacity of individual supported employment projects, even for a limited 
study sample, except as part of a specially-funded research effort. 

The fifth supplementary analysis listed on Figure 8B is the 
collection of data on the benefits and costs to employers hiring 
supported eaployees. Once agax a variety of data would have to be 
collected for a sample of participating employers, including: 

• additional training and supervision costs; 

• other job accommodation costs; 

er|c 
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• costs or cost savings from supported worker productivity 
compared to other workers; 

• costs or cost savings from supported worker absentee- 
ism/job turnover ratef: compared to other workers; and 



• UUSl^ 01 cost savings from supported worker wages and 
fringe benefit-costs compared to other workers* 

These data are not likely to be part of the ongoing data collection 
efforts of a supported employment project, A special research study at 
the state or federal level could be used to generate data on these 
measures. 

The final supplementary analysis which may be of particular 
interest to some projects or some project models is an analysis of how 
total project costs are allocated to various activities, such as general 
administration, outreach, job development, direct participant super— 
vision and/or training, and indirect case management services. The 
measures based on these data would be used to compare the percentage of 
the project budget allocated to different activities, as well as shifts 
in these percentages from one reporting period to another, or from one 
project to another. 
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II. DEVELOPING A DATA COLLECTION STRATEGY AMD IMPLEMENTATION PLAN 

CONSISTENCY VERSJS VARIATION 

By suggesting common data elements and measurei^ that may be used 
across different supported employment projects and/or states, the set of 
core consensus measures represents an attempt to define common 
approaches to measurement rather than mandatory or standardized informa- 
tion systems. Variation or flexibility in data collection practices 
across projects and across states will be expected due to: 1) 
variations in project models, 2) how long the projects have been in 
existence, 3) particular local or state characteristics, and 4)the 
priority given to different objectives. For example, on a local level, 
the data collection and monitoring system for a new free—standing pro- 
ject which uses group work stations in an urban industrial area with 
accessible public transportation and a priority on serving individuals 
from day activity centers may look somewhat different from an informa- 
tion system for a project using an individual placement model based in 
an existing service organization located in a rural area. 

On a state level, data systems will also vary based on whether an 
existing information system is expanded to include supported employment 
variables or whether an entirely new system is developed. The 
utilization of an existing system may provide a state with a broader 
base of participants, but may be more limited in the types of data able 
to be collected, and the timing of that collection. Definitional 
problems may also arise when expanding an existing state system, given 
any changes from historically accepted to current terminology. Other 
states may choose to design and imj'lement an entirely new data system 
especially for supported employment programs. While this may enable 
those states to develop a unique system reflecting the timing, 
definitions and participants involved particularly with supported 
employment, a separate system may be limited by its specific supported 
employment focus rather than on providing more global information about 
the entire potential target population^ Thus, a range of both local and 
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state level data systems have and will continue to emerge along with the 
development and refinement of supported employment programs. 

However, there does appear to be keen interest in potential for 
promoting consistency in data collection where possible, through the 
defining and implementation of a set of broadly applicable measures such 
as the ones presented in Chapter !♦ These may be useful to projects 
for internal uses, including the refinement of service practices and the 
self-monitoring of project development. Additional potential uses of 
standardized data include accountability tc funding agencies and "cross- 
fertilization" through sharing of information between programs which are 
collecting data using the same date elements. The suggested core data 
set also repres'^nts a framework for state systems to consider in 
designing monitoring systems for grantees, in order to include data on 
outcomes the states want to encourage as well as to include measures 
they want all projects to be held accountable for. On a national level, 
these measures represent the building blocks for federally initiated 
evaluation or research about supported employment, and provide a frame- 
work for designing such an effort which addresses the necessity for 
basic uniformity in definitions, data elements, and documentation 
procedures. 

Whether or not a large-scale federal overview of supported employ- 
ment efforts ever occurs, this study's findings are an encouragement for 
states and projects to determine the important questions to be asked 
about the effectiveness of supported employment efforts and the best 
w^s to answer key performance questions. The core or minimum set of 
measures described in this report are measures that are intended to be 
used flexibly by states, with respect to individual service providers. 
Some states may only be able to look toward implementation for programs 
receiving specific supported employment fxinding. Others will be able to 
collect information from all programs offering supported employment 
alternatives regardless of funding source. However states choose to 
implement a system of program monitoring and evaluation, it will be 
critical to carefully document: (1) the range of participants served; 
(2) the accomplishments of the program; (3) the best practices; and 
(A) systems changes with an eye toward generating a rich data base of 
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common information on a cross-project scale as well as the unique 
aspects of particular programs. 



OVERVIEW OF IMPLEMENTATION OPTIONS 



Who Collects the Data and Who Maintains a Client-level Data Base ? 

With the exception of systems change and costs measures, the core 
and supplemental data items and measures described in the previous 
chapter are generally stated in the form of client-level data elements. 
However, state or national level core consensus measures could be con- 
structed from project-level summary reporting measures, rather than 
necessarily requiring the maintenance of a client-level data base at the 
state or national level. 

There are two basic alternatives for collecting client l€vel data 
— either at the project level or at the state level. In either case, 
it is the service provider maintaining the participant files who will 
have ultimate responsibility for initially extracting and recording 
these data. Thus, there are two basic options: 

• The data from the individual client files are reproduced 
and simply passed on to the state for aggregations; or 

• Projects maintain the only set of participant-level 
records due to concerns about: 

— participant confidentiality, 

— familiarity with the information, 
consistency of the data, and/ or 

choice of state to not collect or require reporting 
of client-level data? 



In this last instance, projects would prepare aggregate statistics for 
submission to state-level monit ors/evaluat ors. 

Who Aggregates the Data and Generates Summary Reporting Measures ? 

Figure 9 illustrates options for collecting and reporting data 
describing the project, state and national perspectives on supported 
employment accomplishments. As mentioned previously, a client-level 
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Figure 9 

Data Collection end Reporting Options 



Client-Level Measures 



Option A ; 



Project collects 
data on clients 



Project enters ^ Project generates 

client data into project-level 
project-level MIS statistics 



Project reports 
statistics to 
state 



Option B i 



Project collects 
data on clients 



Project enters 
client data into ^ 
project-level MIS 
for its own use 



Project also pro- 
vides state with 
copy of the client- 
level data base 



State uses this 
client-level data 
base to generate 
summary statistics 
for project 



Option C ; Project collects 
data on clients 



Project provides 
state with paper 
records on each 
client 



State enters data 
into a state-wide 
client-level data 
base 



State uses this 
client-^level data 
base to generate 
summary statistics 
for project 



Pjoject Level Measures 

Project collects 
information on 
project-level 
measures (e.g., 
costs) 



Project provides 
state with this 
information 



System-Level Measures 

State or local system administrator state receives and coordinates 

collects information on system-level systeor-level measures 

measures (e.g., total volume of ^ 
participants participating in supported 
employment) 



EKLC 



55 



data base can be maintained at either the state or the project levels. 
Which way this occurs and how often the data are summarized depends upon 
state and local project negotiations and intended uses of the data* It 
may be more meaningful for the projects' internal use if the summary 
statistics are prepared and reviewed by project staff before the data 
are passed on to the state. If aggregation and preparation of sxunmary 
performance measures occurs at the state levelj however, states can be 
assured of consistency and quality control across projects and can also 
somewhat relieve the burden of data management at the project level. 
Again, who does what and at what level needs to be negotiated among the 
involved actors. System level measures, as shown in Figure 9, will 
need to be generated and analyzed at the state or federal level. 

Timing Issues 

The timing of data collection and reporting activities is another 
important aspect to consider. It seems likely that most projects would 
prepare quarterly reports of the collected information. The timing of 
the preparation of data summaries would vary somewhat, depending on the 
nature of existing information systems and whether client-level or 
project-level data are being used. Thus, reporting measures may be 
collected on an ongoing or monthly basis even though they are likely to 
be reported only quarterly, or, in the case of cost and system change 
measures, annually. 

Some measures may be collected at program entry (for baseline data) 
for each participant and again when changes occur in participant status. 
Collection of data as changes occur would require an ability to manage 
data collection in terms of client time. That is, the ability to trigger 
a data collection activity at any time during the reporting period. In 
some projects, rather than initiating data entries when changes occur, 
information on current client status may be collected at the end of each 
reporting period. 

Who Analyzes the Data ? 

All involved actors in supported emplcyment program efforts will be 
interested in analyzing the data, or portions of the data, for different 
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purposes. Beyond information for internal monitoring and external 
accountability, projects will find the data useful for disseminating 
state-of-the-art information to service providers across projects and 
across different states through conferences, publications, and direct 
sharing of experiences. 

States will analyze information to: 

• look at accountability issues; 

• develop funding criteria; 

• identify barriers and disincentives for employment; 

• develop and refine policies; and 

• design and fund training and technical assistance 
activities. 

On a periodic or voluntary basis, national or cross-state analyses 
would provide information on variation and commonalities among projects 
and states, as well as overall changes in the service delivery system. 
Nationally, some combination of data from other sources, such as labor 
market characteristics, would also be useful as a part of this national 
effort. Finally, at all levels of aggregation, consumers, parents, and 
caregivers will be looking toward this information to assist in informed 
decision— making about services, increased options, and improved quality 
of life for participants. 



data collection system, most projects will be in the start-up phases of 
project implementation. Even the experienced projects will still be 
adjusting to the new mechanics of defining performance data items, and 
collecting and summarizing data on project performance measures. Under 
these circumstances, it is recommended that the initial outcome data 
generated by a supported employment information system be used as 
descriptive information about program experience, and as broad 
indications of whether state projects and the national program initia- 



PSRSPECTIVES ON PROJECT AND STATE ACCOUNTABILITY 



During the initial stages of implementing a supported employment 
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tive are moving in the sired directions, but not as measures of the 
relative effectiveness ^ different projects or program models^ 

Ultimately, after che initial start-up phase of data collection is 
completed, performance measures can become useful for assessing indi- 
vidual project accomplishments as well as national perf ormance» The 
potential assessment-related uses of the proposed data items and per- 
formance measures include (1) ongoing self-assessment by projects them- 
selves; (2) ongoing monitoring of project performance by an external 
funding agency or administrative agency; and (3) summary measures of 
system change and system effectiveness. 

Performance assessment efforts for supported employment will be 
most effective in furthering program objectives if all relevant actors 
agree on which measures should be used to demonstrate program 
accomplishments, as well as on the relative priority among different 
measures of performance. If this level of consensus can be achieved and 
maintained, then any performance assessment exercise will be viewed as a 
w^ to further agreed-upon progrtjn goals. 

Performance assessment becomes dangerous only when the performance 
measures are viewed merely as an externally imposed monitoring tool that 
can be used to reward or sanction projects based on how well they 
perform. The danger in such a situation is that a supported employment 
project might decide to "play it safe," i.e., enroll only participants 
who they know will succeed in supported employment jobs. This would be 
counter to the program goals since the whole concept of supported 
employment is to take a chance on serving individuals who have already 
been rejected as bad risks by all previous employment service providers* 
If accepted as reflecting consensus about program objectives, it is 
hoped that perforiaance measures can have the opposite effect — that of 
encouraging projects to enroll individuals who have not previously had 
the opportunity to work, and of disseminating information about how to 
enable such individuals to succeed in supported employment. 
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APPENDIX A: 



GDRRENT OSERS DEFINITION OF SOPPORTED EMPLOYMENT 

(Developed for National Demonstration Projects) 



DESIGNI^)G SUPPORTED EMPLOYMENT DEMONSTRATION PROJECTS 



Because Supported Employment Demonstration Projects constitute a 
new priority in the Rehabilitation Services Administration's Program of 
Special Projects and Demonstrations for Providing Vocational 
Rehabilitation Services to Severely Handicapped Individuals, this 
additional information is provided to assist applicants in design of 
proposed projects. The information is descriptive, not regulatory. 
Information in the regulations takes precedence if there is any apparent 
conflict. 

The design of a supported employment demonstration project requires 
four steps: 

1 • An analysis and description of the current system of 
ongoing day and vocational services for persons with 
severe disabilities . This analysis should include 
descriptions of the State's administration of the program 
or programs, provide information on the characteristics 
of local services, and show exactly how many service 
recipients are currently engaged in supported employment. 
Additional information on program philosophy, waiting 
lists, current program outcomes and so on. should be 
included. as should a description of how services 
targeted for the supported employment program differ from 
other day and vocational services for persons with 
disabilities in the State. 

2. A description of the desired statewide system of support- 
ed supported e m ploy m ent. This requires a detailed 
analysis of the system of services the State expects to 
have in place at the end of the proposed project. 
Information on anticipated State administration and 
evaluation procedures, characteristics of local services, 
relationships with the business community, and consumer 
outcomes should be included. 
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3 . An analysis of the discrepan c y between the current and 
desired system of services. This analysis should include 
a precise description of the critical features of service 
content, delivery, management, and administration that 
must be addressed to eliminate discrepancy. 

A. Development of project objectives to remove the 
ident if ied discrepancy. These objectives and activities 
should constitute a comprehensive, longitudinal and 
coordinated effoit to move systematically from the 
current to desired system of supported employment 
services. 

It is anticipated that different states will identify different 
issues to be resolved in order to implement statewide supported 
employment programs. Nevertheless, most states will be able to achieve 
lasting statewide change only by developing strategies for: adjusting 
state plans, regulations, and funding and evaluation procedures for day 
services to reflect the characteristics and outcomes of supported 
employment; developing procedures for inter-agency coordination in 
establishing and funding supported employment; expanding the work 
opportunities available to persons engaged in supported employment; 
developing the capacity of existing and new community organisations to 
provide supported employment; building a cadre of staff who are skilled 
in providing supported employment; and informing consumers, parents, 
advocates, employers and others about the nature and purpose of 
supported employment. Naturally the strategies that will be effective 
in resolving these and other issues will be determined by the 
characteristics of the State, the existing services, and employment 
opportunities. 

Both assessing the current service delivery system and projecting 
the desired one require a thorough understanding of supported employment 
and a reliable process for determining when an individtial is or is not 
engaged in supported employment. The definition in the regulations for 



th:* s program establishes four criteria for supported employment. To be 
in supported employment, an individual must be (1) engaged in 
employment, (2) in regular (integrated) work settings, (3) with ongoing 
support, and (A) he or she must experience a disability so severe that 
ongoing support is essential to maintaining employment. 

To help applicants apply these criteria to evaluate their current 
services and define the statewide system that should result from the 
project, we recommend use of the following measures and standards to 
determine whether an individual or program fits the definition of 
supported employment: 

1. Employment . Supported employment is paid employment 
which cannot exist without a regular opportunity to ^crk. 
An individual should be considered to meet the employment 
criterion if he or she engages in paid work for at least 
an average of our hours each day, five days per week or 
another schedule offering at least 20 hours of work per 
week. This standard does not establish a minimum wage or 
productivity level for supported employment. 

2. Int eRrat ion . Work is integrated when it provides 
freqc^nt daily social interactions with people without 
disabilities who are not paid caregivers. Since few 
state or local agencies currently are able to describe 
the extent of integration of individuals in day services, 
we recommend that the following criteria be used to 
estimate the capacity for integration in supported 
employment: an individual's work can be considered 
integrated when he or she works in a place (a) where no 
more than eight people with disabilities work together 
and which is not immediately adjacent to another program 
serving persons with disabilities and (b) whCire persons 
without disabilities who are not paid caregivers are 
present in the work setting or immediate vicinity. 
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For example, an individual who works in a local bank 
creating microfilm records of transactions clearly meets 
the integration criteria for supported employment. So 
do: six individuals with disabilities who work together' 
in an enclave within an electronic factory; a mobile 
janitorial crew that employs five persons with 
disabilities in communit)^ work sites; and a small bakery 
that employs persons with and without disabilities. 

While integration is much more likely when persons with 
disabilities work singly or in email groups among persons 
who are not dieabled, the social interactions necessary 
for integration are also possible in other program sizes. 

Ongoing support . Supported euploy ment exists only when 
ongoing support is provided. An ind ividual should be 
considered to be receiving ongoing support: (a) when 
public funds are available on an ongoing basis to an 
individurl or service pro\'ider who is responsible for 
providing employment support, and (b) when these funds 
are used for intervention?:: directly related to sustaining 
employment. 

Severe disability . Supported einplpyment exists when the 
persons served r^ouire ongoing support and is 
inappropriate for persons who ^ould be better served in 
time-limited preparation programs leading to independent 
employmi^int. The priority for the Supported Employment 
Demonstration Projects is those ivi dividuals : (1) who 
previc:usly have not been ser^/ed or served successfully by 
vocational rehabilitation because of the lack of ongoing 
services needed to sustain employment after time-limited 
rehabilitation se rv ices are completed. With the 
development of supported employment programs in a state, 
however, it is expected that the vocational 
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rehabilitation agency will provide services to these 
individuals that lead to successful closure into 
supported employment; and (2) who are or may be funded 



individuals who fit these two criteria are included on a 
priority basis, a state may also use the Supported 
Employment Demonstration Projects to establish supported 
employment for other groups of individuals whom it 
chooses to fund for ongoing day services. 



for ongoing services in day programs. 



If those 
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